Techniques for Contacting a Teleoperator

ABSTRACT

Techniques and methods for contacting a teleoperator. For instance, while navigating to a location, progress of an autonomous vehicle may stop. This may be caused by the autonomous vehicle yielding to another vehicle, such as at an intersection or when the other vehicle is located along a route of the autonomous vehicle. While yielding to the other vehicle, the autonomous vehicle may determine that the progress has stopped for a threshold amount of time. Additionally, in some circumstances, the autonomous vehicle may determine that the progress not been stopped due to traffic or a traffic light. Based at least in part on the determinations, the autonomous vehicle may send sensor data to the teleoperator, where the sensor data represents an environment for which the autonomous vehicle is located. Additionally, the autonomous vehicle may send an indication of the other vehicle to the teleoperator.

BACKGROUND

An autonomous vehicle may navigate along designated routes or betweenwaypoints. For example, when a control system receives a request from auser device to pick up the user at a location and provide transport to adestination location, the autonomous vehicle may receive, from thecontrol system, instructions to navigate from the pickup location to thedestination location. However, in some circumstances, progress of theautonomous vehicle navigating to the pickup location or the destinationlocation may be stopped, such as by another vehicle. This may causeproblems, such as delaying the autonomous vehicle or causing theautonomous vehicle to block the flow of traffic.

BRIEF DESCRIPTION OF THE DRAWINGS

The detailed description is described with reference to the accompanyingfigures. In the figures, the left-most digit(s) of a reference numberidentifies the figure in which the reference number first appears. Theuse of the same reference numbers in different figures indicates similaror identical components or features.

FIG. 1A is an example environment that includes autonomous vehiclesdetermining when to contact a teleoperator, in accordance withembodiments of the disclosure.

FIG. 1B is the example environment that now includes the autonomousvehicles receiving instructions from the teleoperator, in accordancewith embodiments of the disclosure.

FIG. 2 is an example environment that includes an autonomous vehicledetermining not to contact a teleoperator based at least in part on theautonomous vehicle being stuck in traffic, in accordance withembodiments of the disclosure.

FIG. 3 is an example environment that includes an autonomous vehicledetermining not to contact a teleoperator based at least in part on theprogress of the autonomous vehicle being impeded due to a traffic light,in accordance with embodiments of the disclosure.

FIG. 4 depicts a block diagram of an example system for implementing thetechniques described herein, in accordance with embodiments of thedisclosure.

FIG. 5 is a block diagram of an example teleoperations systemarchitecture, in accordance with embodiments of the disclosure.

FIG. 6 is a first example user interface showing an autonomous vehicleyielding for an object, in accordance with embodiments of thedisclosure.

FIG. 7 is a second example user interface showing an autonomous vehicleyielding for an object, in accordance with embodiments of thedisclosure.

FIG. 8 depicts an example process for determining whether to contact ateleoperator, in accordance with embodiments of the present disclosure.

FIG. 9 depicts an example process for navigating an autonomous vehicleusing instructions from a teleoperator, in accordance with embodimentsof the present disclosure.

DETAILED DESCRIPTION

As discussed above, an autonomous vehicle may navigate along designationroutes or between waypoints, such as by navigating from a pickuplocation to a destination location. However, in some circumstances,progress of the autonomous vehicle may be impeded. For a first example,the autonomous vehicle may store data representing a policy instructingthe autonomous vehicle to yield to other vehicles that arrive at anintersection before the autonomous vehicle. If the autonomous vehicle isyielding to another vehicle at an intersection, where the other vehicleis not moving for one or more reasons (e.g., the other vehicle cannotnavigate along a route until the autonomous vehicle moves due to anobstruction or narrow road), then the progress of the autonomous vehiclemay be impeded. For a second example, the autonomous vehicle may set aninteraction associated with another vehicle, where the interactioncauses the autonomous vehicle to follow the other vehicle. If the othervehicle stops, such as by parking or breaking down, then the progress ofthe autonomous vehicle may be impeded.

As such, this application describes techniques for determining when tocontact a teleoperator as well as techniques for navigating theautonomous vehicle using instructions that are received from theteleoperator. For example, when progress of the autonomous vehicle isimpeded, the autonomous vehicle may determine an amount of time that theprogress has been impeded. The autonomous vehicle may then determinewhether the amount of time satisfies (e.g., is equal to or greater than)a threshold amount of time. The threshold amount of time may include,but is not limited to, three seconds, thirty seconds, one minute, threeminutes, and/or any other period of time. If the autonomous vehicledetermines that the amount of time does not satisfy (e.g., is below) thethreshold amount of time, then the autonomous vehicle may determine notto contact the teleoperator. However, if the autonomous vehicledetermines that the amount of time satisfies the threshold amount oftime, then the autonomous vehicle may determine if the progress of theautonomous vehicle is impeded and/or the progress of the object isimpeded due to an intervening condition.

For a first example, the autonomous vehicle may determine if theprogress of the autonomous vehicle is impeded because of traffic. Insome instances, the autonomous vehicle may make the determination basedat least in part on determining whether progress of another vehicle, inaddition to the vehicle that the autonomous vehicle is yielding to, isalso impeded. For instance, when the autonomous vehicle is navigatingalong a multilane road, the autonomous vehicle may determine if progressof another vehicle in another lane has also impeded. If the autonomousvehicle determines that the progress of another vehicle has alsoimpeded, then the autonomous vehicle may determine that traffic hascaused the progress of the autonomous vehicle to be impeded. As such,the autonomous vehicle may determine not to contact the teleoperator.However, if the autonomous vehicle determines that the progress ofanother vehicle has not been impeded and/or there is not another vehiclenext to the autonomous vehicle, then the autonomous vehicle maydetermine that traffic has not caused the progress of the autonomousvehicle to be impeded. As such, the autonomous vehicle may determine tocontact the teleoperator.

For a second example, the autonomous vehicle may determine if theprogress of the autonomous vehicle is impeded because of a trafficlight. In some instances, the autonomous vehicle may use sensor data todetermine if a traffic light is causing the progress of the autonomousvehicle to be impeded. Additionally, or alternatively, in someinstances, the autonomous vehicle may use map data to determine if atraffic light is causing the progress of the autonomous vehicle to beimpeded. In either instance, if the autonomous vehicle determines that atraffic light is causing the progress of the autonomous vehicle to beimpeded, then the autonomous vehicle may determine not to contact theteleoperator. However, if the autonomous vehicle determines that atraffic light is not causing the progress of the autonomous vehicle tobe impeded, then the autonomous vehicle may determine to contact theteleoperator.

When determining to contact the teleoperator, the autonomous vehicle maysend sensor data to one or more computing devices associated with theteleoperator. The sensor data may include, but is not limited to, imagedata, light detection and ranging (lidar) data, radar data, and/or othertypes of sensor data representing the environment around the autonomousvehicle. In some instances, the autonomous vehicle may further send, tothe one or more computing devices, data indicating which object(s)(e.g., vehicle(s), pedestrians, signs, construction equipment, etc.) arecausing the progress of the autonomous vehicle to be impeded. Forexample, if the autonomous vehicle is yielding to another vehicle, thedata may indicate that the autonomous vehicle is currently yielding tothe other vehicle. In some instances, the autonomous vehicle may send,to the one or more computing devices, data indicating the amount of timethat the progress of the autonomous vehicle has been impeded and/or theamount of time that the autonomous vehicle has been yielding to theobject(s).

The one or more computing devices may receive the sensor data from theautonomous vehicle and, in response, display a user interface thatincludes content (e.g., image(s), video, etc.) represented by the sensordata. Additionally, the one or more computing devices may displaygraphical elements indicating which object(s) are causing the progressof the autonomous vehicle to be impeded. For example, the user interfacemay display a graphical element that indicates that the autonomousvehicle is yielding to an object. The teleoperator may then use the oneor more computing devices to send instructions to the autonomousvehicle, where the instructions indicate how the autonomous vehicle isto proceed. In at least some examples, such instructions may comprise aninstruction to disregard and/or alter a policy (e.g., disregard theyield instruction to a selected object in the environment).

For a first example, and if the autonomous vehicle is yielding toobject(s) at an intersection, where the object(s) are not moving for oneor more reasons, then the instructions may cause the autonomous vehicleto no longer yield to the object(s) and/or cause the autonomous vehicleto temporarily cease following (and/or alter) one or more policies. Theone or more policies may include, but are not limited to, a first policyto yield to vehicles that arrive at an intersection before theautonomous vehicle, a second policy to travel in a road lane, and/or oneor more additional policies. In some instances, temporarily ceasingfollowing the one or more policies may include ceasing following the oneor more policies for a period of time. The period of time may include,but is not limited to, five seconds, ten seconds, thirty seconds, oneminute, and/or another period of time. Additionally, or alternatively,in some instances, temporarily ceasing following the one or morepolicies may include ceasing following the one or more policies untilthe autonomous vehicle navigates to a given location, navigates a givendistance, navigates to a location at which the autonomous vehicle wouldno longer be yielding to the object(s), and/or the like. Regardless,using the instructions, the autonomous vehicle may determine to nolonger yield to the object(s) and/or determine to temporarily ceasefollowing the one or more policies. As such, the autonomous vehicle mayagain begin navigating to a location.

For a second example, and if the autonomous vehicle is yielding toanother vehicle for which the autonomous vehicle has a currentinteraction, such as “follow”, then the instruction may cause theautonomous vehicle to change the interaction with the other vehicle. Forinstance, the instruction may cause the interaction to change from“follow” to “do not follow”. Additionally, in some instances, theinstruction may cause the autonomous vehicle to temporarily ceasefollowing one or more polices. For instance, the instructions may causethe autonomous vehicle to temporarily cease following the second policy,which includes traveling in a road lane, so that the autonomous vehiclecan navigate around the other vehicle. Using the instructions, theautonomous vehicle may change the interaction and/or determine totemporarily cease following the one or more policies. As such, theautonomous vehicle may again begin navigating to a location.

In some instances, once the autonomous vehicle begins navigating afterreceiving the instructions, the autonomous vehicle may determine to onceagain yield to the object(s). For instance, the autonomous vehicle maydetermine to once again follow the one or more policies or change theinteraction with the object(s) back to the original interaction setting.The autonomous vehicle may make the determination based at least in parton a movement of the object(s).

For a first example, the autonomous vehicle may determine that avelocity associated the object(s) has changed. For instance, thevelocity may change from zero miles per hour to a velocity that isgreater than zero miles per hour. Based at least in part on the changein the velocity, the autonomous vehicle may determine that the object(s)are starting to move. As such, the autonomous vehicle may determine toonce again yield to the object(s) and/or change the interaction with theobject(s) back to “follow”. Additionally, in some instances, theautonomous vehicle may again contact the teleoperator.

For a second example, the object(s) may have originally been moving whenthe autonomous vehicle originally yielded to the object(s), however, theobject(s) may have remained in a given area. For instance, the object(s)may include a construction vehicle (e.g., backhoe, paver, etc.) that ismoving back-and-forth within the given area. In such an example, theinstructions sent by the teleoperator may include a specified areaaround the object(s) for which the object(s) are expected to remain. Ifthe object(s) move outside of the specified area, then the autonomousvehicle may determine to once again yield to the object(s).Additionally, the autonomous vehicle may once again contact theteleoperator.

While the examples described above include determining that the progressof the autonomous vehicle is impeded using policies and/or interactions,in other instances, the progress of the autonomous vehicle may beimpeded based on one or more other factors. For a first example, theautonomous vehicle may determine that the progress is impeded based onidentifying a specific type of object. For instance, the specific typeof object may include a tractor and/or streetcleaner that navigates at aslow velocity. As such, the progress of the autonomous vehicle may beimpeded when the specific type of object is located along the route ofthe autonomous vehicle. For a second example, the autonomous vehicle maydetermine that the progress is impeded based on identifying a specifictype of scenario. The scenario may include, but is not limited to,another vehicle merging into the drive lane of the autonomous vehicle,another vehicle navigating at a velocity that is below a thresholdvelocity (which is described below), an accident that occurs along theroute of the autonomous vehicle, and/or the like.

The examples above describe the autonomous vehicle as determining if“progress” of the autonomous vehicle has been impeded. In someinstances, the autonomous vehicle may determine that the progress hasbeen impeded based at least in part on determining that the autonomousvehicle has stopped at a location (e.g., the velocity of the autonomousvehicle is zero miles per hour). Additionally, or alternatively, in someinstances, the autonomous vehicle may determine that the progress hasbeen impeded based at least in part on determining that the autonomousvehicle is only navigating a give distance within a given amount oftime. For example, the autonomous vehicle may determine that theprogress has been impeded based at least in part on determining that theautonomous vehicle is navigating one foot per second, one foot per fiveseconds, one foot per minute, and/or the like. Additionally, oralternatively, in some instances, the autonomous vehicle may determinethat the progress has been impeded based at least in part on determiningthat the current velocity of the autonomous vehicle is below a thresholdvelocity. The threshold velocity may include, but is not limited to, ahalf a mile per hour, a mile per hour, and/or any other velocity.

Additionally, the techniques may provide the teleoperator withcontextual information of the situation at the autonomous vehicle sothat the teleoperator may provide guidance to the autonomous vehicle. Ateleoperations system may include one or more teleoperators, which maybe human teleoperators, located at a teleoperations center. In someexamples, one or more of the teleoperators may not be human, such as,for example, they may be computer systems leveraging artificialintelligence, machine learning, and/or other decision-making strategies.In some examples, the teleoperator may interact with one or moreautonomous vehicles in a fleet of autonomous vehicles via a userinterface that can include a teleoperator interface. The teleoperatorinterface may include one or more displays configured to provide theteleoperator with data related to operations of the autonomous vehicles.For example, the display(s) may be configured to show content related tosensor data received from the autonomous vehicles, content related tothe road network, and/or additional content or information to facilitateproviding assistance to the autonomous vehicles.

Additionally, or alternatively, the teleoperator interface may alsoinclude a teleoperator input device configured to allow the teleoperatorto provide information to one or more of the autonomous vehicles, forexample, in the form of teleoperation instructions providing guidance tothe autonomous vehicles. The teleoperator input device may include oneor more of a touch-sensitive screen, a stylus, a mouse, a dial, akeypad, a microphone, a touchscreen, and/or a gesture-input systemconfigured to translate gestures performed by the teleoperator intoinput commands for the teleoperator interface. As explained in moredetail herein, the teleoperations system may provide one or more of theautonomous vehicles with guidance to avoid, maneuver around, or passthrough events, such as when progress of the autonomous vehicles hasbeen impeded.

The techniques described herein can be implemented in a number of ways.Example implementations are provided below with reference to thefollowing figures. Although discussed in the context of a vehicle, themethods, apparatuses, and systems described herein can be applied to avariety of systems.

FIG. 1A is an example environment 100 that includes autonomous vehicles102(1)-(2) determining when to contact a teleoperator, in accordancewith embodiments of the disclosure. For example, the first autonomousvehicle 102(1) may be navigating along a route 104. While navigating,the first autonomous vehicle 102(1) may encounter a vehicle 106 that isalso navigating along the route 104. As such, the first autonomousvehicle 102(1) may use sensor data 108 generated by one or more sensorsof the first autonomous vehicle 102(1) to identify the vehicle 106.Additionally, since the vehicle 106 is located in front of the firstautonomous vehicle 102(1), and in the same driving lane as the firstautonomous vehicle 102(1), the first autonomous vehicle 102(1) may setan interaction with the vehicle 106 as “follow”, which is represented bythe interaction data 110.

However, as the first autonomous vehicle 102(1) is navigating along theroute 104 and following the vehicle 106, the vehicle 106 may stop forone or more reasons. For example, the vehicle 106 may breakdown, causingthe vehicle 106 to at least partially obstruct the road, as illustratedin the example of FIG. 1A. As such, and since the first autonomousvehicle 102(1) is following the vehicle 106, the first autonomousvehicle 102(1) may also stop at a location 112 along the route 104. Thismay cause the first autonomous vehicle 102(1) to determine that theprogress of the first autonomous vehicle 102(1) has been impeded.

While the progress is impeded, the first autonomous vehicle 102(1) maydetermine that an amount of time at which the progress has been impededsatisfies a threshold amount of time. The first autonomous vehicle102(1) may further determine if the progress has been impeded due to anintervening condition (e.g., traffic, a traffic light, etc.). In theexample of FIG. 1A, the first autonomous vehicle 102(1) may determinethat the progress has not been impeded due to an intervening condition.Since the amount of time satisfies the threshold amount of time, andsince the progress has not been impeded due to an intervening condition,the first autonomous vehicle 102(1) may determine to contact theteleoperator.

To contact the teleoperator, the first autonomous vehicle 102(1) maysend, to a teleoperator system 114 associated with the teleoperator, atleast a portion of the sensor data 108. For instance, the firstautonomous vehicle 102(1) may send at least sensor data 108 (e.g., imagedata) representing the vehicle 106 and/or any derived data therefrom(e.g., detections, classifications, control data, etc.) to theteleoperator system 114. In some instances, the first autonomous vehicle102(1) may further send, to the teleoperator system 114, indicator data116 that indicates that the progress has been impeded due to the vehicle106 and/or that the current interaction with the vehicle 106 includes“follow”. The teleoperator may use the data received from the firstautonomous vehicle 102(1) to provide guidance to the first autonomousvehicle 102(1), which is described in the example of FIG. 1B.

Additionally, in the example of FIG. 1A, the second autonomous vehicle102(2) may be navigating along a route 118. While navigating, the secondautonomous vehicle 102(2) may encounter an intersection, which causesthe second autonomous vehicle 102(2) to stop at a location 120.Additionally, the second autonomous vehicle 102(2) may use sensor data122 generated by one or more sensors of the second autonomous vehicle102(2) to identify that the vehicle 124 arrived at the intersectionbefore the second autonomous vehicle 102(2). As such, and based at leastin part on a policy represented by the policy data 126, the secondautonomous vehicle 102(2) may determine to yield to the vehicle 124 atthe intersection.

However, the vehicle 124, which is navigating along a route 128, may beunable to advance along the route 128 until after the second autonomousvehicle 102(2) navigates along the route 118. This is because, based atleast in part on a width 130 of the second autonomous vehicle 102(2) anda width 132 of a drive envelope along the route 128, the secondautonomous vehicle 102(2) is at least partially obstructing the route128 of the vehicle 124. As such, the vehicle 124 may continue to waitfor the second autonomous vehicle 102(2) to navigate along the route 118before the vehicle 124 navigates along the route 128. Since the secondautonomous vehicle 102(2) is yielding to the vehicle 124, the progressof the second autonomous vehicle 102(2) may be impeded at the location120.

While the progress is impeded, the second autonomous vehicle 102(2) maydetermine that an amount of time at which the progress has been impededsatisfies a threshold amount of time. The second autonomous vehicle102(2) may further determine if the progress has been impeded due to anintervening condition (e.g., traffic, a traffic light, etc.). In theexample of FIG. 1A, the second autonomous vehicle 102(2) may determinethat the progress has not been impeded due to an intervening condition.Since the amount of time satisfies the threshold amount of time, andsince the progress is has not been impeded due to an interveningcondition, the second autonomous vehicle 102(2) may determine to contactthe teleoperator.

To contact the teleoperator, the second autonomous vehicle 102(2) maysend, to the teleoperator system 114 associated with the teleoperator,at least a portion of the sensor data 122 (and/or any data derivedtherefrom). For instance, the second autonomous vehicle 102(2) may sendat least sensor data 122 (e.g., image data) representing the vehicle 124to the teleoperator system 114. In some instances, the second autonomousvehicle 102(2) may further send, to the teleoperator system 114,indicator data 134 that indicates that the progress has been impeded dueto the vehicle 124 and/or that the second autonomous vehicle 102(2) iscurrently yielding to the vehicle 124. Still, in some instances, thesecond autonomous vehicle 102(2) may send, to the teleoperator system114, the policy data 126 indicating the policy that is causing thesecond autonomous vehicle 102(2) to yield to the vehicle 124. Theteleoperator may then use the data received from the second autonomousvehicle 102(2) to provide guidance to the second autonomous vehicle102(2), which is described in the example of FIG. 1B.

FIG. 1B is the example environment 100 that now includes the autonomousvehicles 102(1)-(2) receiving instructions from the teleoperator, inaccordance with embodiments of the disclosure. For example, theteleoperator may interact with the autonomous vehicle 102(1)-(2) via auser interface that can include a teleoperator interface. Theteleoperator interface may include one or more displays configured toprovide the teleoperator with data related to operation of theautonomous vehicles 102(1)-(2). For example, the display(s) may beconfigured to show content (e.g., image(s)) related to the sensor data108 received from the first autonomous vehicle 102(1). Additionally, thedisplay(s) may be configured to shown content (e.g., image(s)) relatedto the sensor data 122 received from the second autonomous vehicle102(2).

In some instances, the display(s) may further be configured to showcontent related to object(s) that are causing progresses of theautonomous vehicles 102(1)-(2) to be impeded. For example, thedisplay(s) may use the indicator data 116 in order to show a graphicalelement indicating that it is the vehicle 106 that is causing theprogress of the first autonomous vehicle 102(1) to be impeded.Additionally, the display(s) may use the indicator data 134 to show agraphical element indicating that it is the vehicle 124 that is causingthe progress of the second autonomous vehicle 102(2) to be impeded. Asdescribed herein, a graphical element may include, but is not limitedto, a bounding box around an object, shading an object, text describingthe object, and/or any other graphical element that may indicateinformation about the object.

The teleoperator interface may also include a teleoperator input deviceconfigured to allow the teleoperator to provide information to theautonomous vehicles 102(1)-(2), for example, in the form ofteleoperation instructions providing guidance to the autonomous vehicles102(1)-(2). For example, the teleoperator may provide instructions,represented by the instruction data 136, for the first autonomousvehicle 102(1). The instructions may indicate that the first autonomousvehicle 102(1) is to change the interaction with the vehicle 106 from“follow” to “do not follow”. In some instances, the instructions mayfurther indicate that the first autonomous vehicle 102(1) maytemporarily cease following one or more policies (and/or temporarilyalter such policies), such as a policy specifying that the firstautonomous vehicle 102(1) is to maintain a road lane, in order for thefirst autonomous vehicle 102(1) to navigate around the vehicle 106.

The first autonomous vehicle 102(1) may receive the instruction data 136from the teleoperator system 114. Using the instruction data 136, thefirst autonomous vehicle 102(1) may update the interaction with thevehicle 106 to indicate “do not follow”. Additionally, in someinstances, the first autonomous vehicle 102(1) may temporarily ceasefollowing (or otherwise alter) the one or more policies. As such, andbased at least in part on the instructions, the first autonomous vehicle102(1) may determine a route 138 to get around the vehicle 106. Aftergetting around the vehicle 106, the first autonomous vehicle 102(1) mayagain navigate along the route 104 of the first autonomous vehicle102(1).

In some instances, while navigating along the route 138 to get aroundthe vehicle 106, the first autonomous vehicle 102(1) may determine ifthe vehicle 106 begins moving. For example, the first autonomous vehicle102(1) may use the sensor data 108 to determine if a velocity for thevehicle 106 changes from zero miles per hour to a velocity that isgreater than zero miles per hour. In some instances, if the firstautonomous vehicle 102(1) determines that the vehicle 106 begins moving,the first autonomous vehicle 102(1) may stop navigating along the route138. Additionally, the first autonomous vehicle 102(1) may again updatethe interaction with the vehicle 106 to include “follow,” therebyreturning to the vehicle's nominal operating behavior.

Additionally, or alternatively, in some instances, the instructions fromthe teleoperator system 114 may indicate an area 140 around the vehicle106. In such instances, while navigating along the route 138 to getaround the vehicle 106, the first autonomous vehicle 102(1) maydetermine if the vehicle 106 moves outside of the area 140. For example,the first autonomous vehicle 102(1) may use the sensor data 108 todetermine if at least a portion of the vehicle 106 moves to outside ofthe area 140. If the first autonomous vehicle 102(1) determines that thevehicle 106 moves outside of the area 140, the first autonomous vehicle102(1) may stop navigating along the route 138. Additionally, the firstautonomous vehicle 102(1) may again update the interaction with thevehicle 106 to include “follow”.

Additionally, the teleoperator may provide instructions, represented bythe instruction data 142, for the second autonomous vehicle 102(2). Theinstructions may indicate that the second autonomous vehicle 102(2) isto cease yielding for the vehicle 124. In some instances, theinstructions may further indicate that the second autonomous vehicle102(2) may temporarily cease following (or otherwise alter) one or morepolicies, such as a policy specifying that the second autonomous vehicle102(2) is to yield to vehicles that arrive at intersections before thesecond autonomous vehicle 102(2).

The second autonomous vehicle 102(2) may receive the instruction data142 from the teleoperator system 114. Using the instruction data 142,the second autonomous vehicle 102(2) may determine to no longer yieldfor the vehicle 124 at the intersection. Additionally, in someinstances, the second autonomous vehicle 102(2) may temporarily ceasefollowing the one or more policies. As such, and based at least in parton the instructions, the second autonomous vehicle 102(2) may beginnavigating along the route 118 before the vehicle 124 begins navigatingalong the route 128 in order to provide the vehicle 124 with room tonavigate along the route 128. As shown in the example of FIG. 1B, afterthe second autonomous vehicle 102(2) begins to navigate along the route118, the vehicle 124 may then begin to navigate along the route 128.

While the example of FIG. 1A describes the progress of the firstautonomous vehicle 102(1) as being impeded based on the vehicle 106,which the first autonomous vehicle 102(1) is following stopping, inother examples, the progress of the first autonomous vehicle 102(1) maystop for other reasons. For example, first autonomous vehicle 102(1) mayanalyze the sensor data 108 and, based on the analysis, the firstautonomous vehicle 102(1) may determine that the vehicle 106 includes aspecific type of object. Based on the determination, the firstautonomous vehicle 102(1) may determine that the progress is impeded.For a second example, the first autonomous vehicle 102(1) may analyzethe sensor data 108 and, based on the analysis, the first autonomousvehicle 102(1) may determine that a velocity of the vehicle 106 is belowa threshold velocity. Based on the determination, the first autonomousvehicle 102(1) may determine that the progress is impeded.

FIG. 2 is an example environment 200 that includes an autonomous vehicle202 determining not to contact a teleoperator based at least in part onthe autonomous vehicle 202 being stuck in traffic, in accordance withembodiments of the disclosure. For instance, the autonomous vehicle 202may be navigating along a route 204 on a road that includes more thanone drive lane. In some instances, the autonomous vehicle 202 may usesensor data and/or map data to determine that the road includes morethan two drive lanes. For a first example, the autonomous vehicle 202may analyze the sensor data to identify marking on the road thatindicate multiple drive lanes in the same direction. For a secondexamples, the autonomous vehicle 202 may analyze the sensor data toidentify another vehicle navigating next to the autonomous vehicle 202and in a same direction as the autonomous vehicle 202. For a thirdexample, the map data may indicate that the road includes multiple drivelanes.

Additionally, other vehicles 206(1)-(11) may respectively be navigatingalong routes 208(1)-(11). While navigating, the autonomous vehicle 202and the vehicles 206(1)-(11) may get stuck in traffic. For instance, theprogress of the autonomous vehicle 202 may be impeded at a locationalong the route 204. Based at least in part on the progress beingimpeded, the autonomous vehicle 202 may determine whether to contact theteleoperator.

For instance, the autonomous vehicle 202 may determine that the progresshas been impeded for a threshold amount of time. The autonomous vehicle202 may then determine if the progress is impeded due to traffic. Insome instances, to determine if the progress has been impeded due totraffic, the autonomous vehicle 202 may determine that the progress forat least one other vehicle located near the autonomous vehicle 202and/or located near the vehicle 206(10) for which the autonomous vehicle202 is following has also impeded.

For example, the autonomous vehicle 202 may be navigating in a firstroad lane 210(1). The autonomous vehicle 202 may determine that theprogress of the vehicle 206(2) located in a second road lane 210(2), theprogress of the vehicle 206(6) located in the second road lane 210(2),the progress of the vehicle 206(9) located in the second road lane210(2), the progress of the vehicle 206(4) located in a third road lane210(3), the progress of the vehicle 206(7) located in the third roadlane 210(3), and/or the progress of the vehicle 206(11) located in thethird road lane 210(3) has also been impeded. Based at least in part onthe determination(s), the autonomous vehicle 202 may determine that theprogress is impeded due to traffic. As such, the autonomous vehicle 202may determine not to contact the teleoperator even though the progressof the autonomous vehicle 202 has been impeded for the threshold amountof time.

FIG. 3 is an example environment 300 that includes an autonomous vehicle302 determining not to contact a teleoperator based at least in part onthe progress of the autonomous vehicle 302 being impeded due to atraffic light 304, in accordance with embodiments of the disclosure. Forinstance, the autonomous vehicle 302 may be navigating along a route306. Additionally, other vehicles 308(1)-(4) may respectively benavigating along routes 310(1)-(4). While navigating, the progress ofthe autonomous vehicle 302 may be impeded based at least in part on theautonomous vehicle 302 yielding to the vehicle 308(1), which alsostopped. For instance, the vehicle 308(1) may be stopped because thetraffic light 304 is red. Based at least in part on the progress beingimpeded, the autonomous vehicle 302 may determine whether to contact theteleoperator.

For instance, the autonomous vehicle 302 may determine that the progresshas been impeded for a threshold amount of time. The autonomous vehicle302 may then determine if the progress is impeded due to the trafficlight 304. In some instances, to determine if the progress is stoppeddue to the traffic light 304, the autonomous vehicle 302 may analyzesensor data generated by the autonomous vehicle 302. Based at least inpart on the analysis, the autonomous vehicle 302 may identify thetraffic light 304 and determine that the traffic light 304 is currentlyred. As such, the autonomous vehicle 302 may determine that the progressis impeded due to the traffic light 304. In some instances, since theprogress is impeded due to the traffic light 304, the autonomous vehicle302 may determine not to contact the teleoperator even though theprogress has been impeded for the threshold amount of time.

In a similar example, however, if such an environment 300 is devoid oftraffic lights, the autonomous vehicle 302 may reason about how toproceed based on common practice or law dictating how to proceed througha four way stop (e.g., yield based on order of approach and/or to thevehicle to the right). In such an example, if the autonomous vehicle 302has been yielding (according to the policy) to another vehicle (e.g.,vehicle 308(4)) for a period of time greater than or equal to athreshold amount of time, the autonomous vehicle 302 may send a signalto a teleoperator for guidance.

FIG. 4 depicts a block diagram of an example system 400 for implementingthe techniques described herein, in accordance with embodiments of thedisclosure. In at least one example, the system 400 can include avehicle 402 (which may represent, and/or be similar to, the firstautonomous vehicle 102(1), the second autonomous vehicle 102(2), theautonomous vehicle 202, and/or the autonomous vehicle 302). The vehicle402 can include a vehicle computing device 404, one or more sensorsystems 406, one or more emitters 408, one or more communicationconnections 410, at least one direct connection 412, and one or moredrive modules 414.

The vehicle computing device 404 can include one or more processors 416and a memory 418 communicatively coupled with the one or more processors416. In the illustrated example, the vehicle 402 is an autonomousvehicle. However, the vehicle 402 may be any other type of vehicle(e.g., a manually driven vehicle, a semi-autonomous vehicle, etc.), orany other system having at least an image capture device. In theillustrated example, the memory 418 of the vehicle computing device 404stores a localization component 420, a perception component 422, aplanning component 424, a progress component 426, an conditionscomponent 428, a teleoperator component 430, one or more systemcontrollers 432, and one or more maps 434. Though depicted in FIG. 4 asresiding in the memory 418 for illustrative purposes, it is contemplatedthat the localization component 420, the perception component 422, theplanning component 424, the progress component 426, the conditionscomponent 428, the teleoperator component 430, the one or more systemcontrollers 432, and/or the one or more maps 434 can additionally, oralternatively, be accessible to the vehicle 402 (e.g., stored on, orotherwise accessible by, memory remote from the vehicle 402).

In at least one example, the localization component 420 can includefunctionality to receive sensor data 436 (which may represent, and/or besimilar to, the sensor data 108 and/or the sensor data 122) from thesensor system(s) 406 and to determine a position and/or orientation ofthe vehicle 402 (e.g., one or more of an x-, y-, z-position, roll,pitch, or yaw). For example, the localization component 420 can includeand/or request/receive a map of an environment and can continuouslydetermine a location and/or orientation of the vehicle 402 within themap. In some instances, the localization component 420 can utilize SLAM(simultaneous localization and mapping), CLAMS (calibration,localization and mapping, simultaneously), relative SLAM, bundleadjustment, non-linear least squares optimization, or the like toreceive image data, lidar data, radar data, IMU data, GPS data, wheelencoder data, and the like to accurately determine a location of thevehicle 402. In some instances, the localization component 420 canprovide data to various components of the vehicle 402 to determine aninitial position of the vehicle 402 for generating a candidatetrajectory, as discussed herein.

In some instances, the perception component 422 can includefunctionality to perform object detection, segmentation, and/orclassification. In some instances, the perception component 422 canprovide processed sensor data 436 that indicates a presence of an objectthat is proximate to the vehicle 402 and/or a classification of theobject as an object type (e.g., car, pedestrian, cyclist, animal,building, tree, road surface, curb, sidewalk, unknown, etc.). Inadditional and/or alternative examples, the perception component 422 canprovide processed sensor data 436 that indicates one or morecharacteristics associated with a detected object and/or the environmentin which the object is positioned. In some instances, characteristicsassociated with an object can include, but are not limited to, anx-position (global position), a y-position (global position), az-position (global position), an orientation (e.g., a roll, pitch, yaw),an object type (e.g., a classification), a velocity of the object, anacceleration of the object, an extent of the object (size), etc.Characteristics associated with the environment can include, but are notlimited to, a presence of another object in the environment, a state ofanother object in the environment, a time of day, a day of a week, aseason, a weather condition, an indication of darkness/light, etc.

In general, the planning component 424 can determine a path for thevehicle 402 to follow to traverse through an environment. For example,the planning component 424 can determine various routes and trajectoriesand various levels of detail. For example, the planning component 424can determine a route to travel from a first location (e.g., a currentlocation) to a second location (e.g., a target location). For thepurpose of this discussion, a route can be a sequence of waypoints fortravelling between two locations. As non-limiting examples, waypointsinclude streets, intersections, global positioning system (GPS)coordinates, etc. Further, the planning component 424 can generate aninstruction for guiding the vehicle 402 along at least a portion of theroute from the first location to the second location. In at least oneexample, the planning component 424 can determine how to guide thevehicle 402 from a first waypoint in the sequence of waypoints to asecond waypoint in the sequence of waypoints. In some instances, theinstruction can be a trajectory, or a portion of a trajectory. In someinstances, multiple trajectories can be substantially simultaneouslygenerated (e.g., within technical tolerances) in accordance with areceding horizon technique, wherein one of the multiple trajectories isselected for the vehicle 402 to navigate.

In at least one example, the planning component 424 can determine apickup location associated with a location. As used herein, a pickuplocation can be a specific location (e.g., a parking space, a loadingzone, a portion of a ground surface, etc.) within a threshold distanceof a location (e.g., an address or location associated with a dispatchrequest) where the vehicle 402 can stop to pick up a passenger. In atleast one example, the planning component 424 can determine a pickuplocation based at least in part on determining a user identity (e.g.,determined via image recognition or received as an indication from auser device, as discussed herein). Arrival at a pickup location, arrivalat a destination location, entry of the vehicle by a passenger, andreceipt of a “start ride” command are additional examples of events thatmay be used for event-based data logging.

In some instances, the planning component 424 can further includefunctionality to determine when the vehicle 402 is to yield toobject(s), such as other vehicle(s). For instance, the planningcomponent 424 may use policy data 438 to determine when the vehicle 402is to yield to the object(s). The policy data 438 may represent one ormore policies that the vehicle 402 is to follow when navigating. The oneor more policies may include, but are not limited to, a policy to yieldto other vehicle(s) that arrive at an intersection before the vehicle402, a policy to travel in a road lane, a policy on how to change lanes,a policy on how to stop, a policy that the vehicle 402 should nottailgate, and/or the like. As such, the planning component 424 maydetermine that the vehicle 402 is to yield to another vehicle when theother vehicle arrives at an intersection before the vehicle 402.Additionally, the planning component 424 may determine that the vehicleis to yield to two other vehicles when the two other vehicles arrive atan intersection before the vehicle 402.

In some instances, the planning component 424 can further includefunctionality to determine an interaction between the vehicle 402 andobject(s), such as vehicle(s). For instance, the planning component 424may use interaction data 440 to determine the interaction between thevehicle 402 and the object(s). The interaction data 440 may representone or more interactions, such as, but not limited to, an interaction tofollow object(s) that are located along a trajectory of the vehicle 402,an interaction not to follow object(s) that are not located along atrajectory of the vehicle 402, an interaction to yield to object(s) forwhich the vehicle 402 is following, and/or the like. In some instances,following may involve varying longitudinal controls of the vehicle 402(e.g., velocity and/or acceleration) to maintain a minimum distancebetween the vehicle 402 and the object, wherein the minimum distance maybe based, at least in part, on a relative distance and/or velocitybetween the vehicle 402 and the object. As such, the planning component424 may determine that the vehicle 402 is to follow another vehicle whenthe other vehicle is located in the trajectory of the vehicle 402.

In some instances, the planning component 424 can further includefunctionality to generate indicator data 442, where the indicator data442 represents object(s) that the vehicle 402 is currently yielding toand/or object(s) that the vehicle 402 is following. In some instances,the planning component 424 may generate the indicator data 442 each timethe perception component determines that the vehicle 402 is yielding toobject(s) and/or each time the planning component 424 determines thatthe vehicle 402 is following object(s). In other instances, the planningcomponent 424 may generate the indicator data 442 when the vehicle 402determines to contact the teleoperator.

In some instances, the planning component 424 includes functionality touse instruction data 446 sent by the teleoperator system 114 in order todetermine how to navigate the vehicle 402. For a first example, theinstruction data 446 may represent an instruction to cease yielding toobject(s). Based at least in part on the instruction data 446, theplanning component 424 may cause the vehicle 402 to cease yielding tothe object(s) and/or determine a route for navigating the vehicle 402.For a second example, the instruction data 446 may represent aninstruction to temporarily cease following one or more policies. Basedat least in part on the instruction data 446, the planning component 424may cause the vehicle 402 to temporarily cease following the one or morepolicies and/or determine a route for navigating the vehicle 402. For athird example, the instruction data 446 may represent an instruction tochange an interaction with object(s) from “follow” to “no longerfollow”. Based at least in part on the instruction data 446, theplanning component 424 may cause the vehicle 402 to change theinteraction with the object(s) and/or determine a route for navigatingaround the object(s).

The progress component 426 can include functionality to determine whenthe progress of the vehicle 402 has been impeded. In some instances, theprogress component 426 may determine that the progress of the vehicle402 has been impeded when the vehicle 402 is not moving (e.g., avelocity of the vehicle 402 is zero miles per hour). Additionally, oralternatively, in some instances, the progress component 426 maydetermine that the progress of the vehicle 402 has been impeded when thevehicle 402 only navigates a give distance within a given amount oftime. Additionally, or alternatively, in some instances, the progresscomponent 426 may determine that the progress of the vehicle 402 hasbeen impeded when the current velocity of the vehicle 402 is below athreshold velocity.

The progress component 426 can include functionality to determine whenthe progress of the vehicle 402 is impeded by an object. In someinstances, the progress component 426 may determine that the progress ofthe vehicle 402 may be impeded when the vehicle 402 identifies aspecific type of object located within the environment. In someinstances, the progress component 426 may determine that the progress ofthe vehicle 402 may be impeded when the vehicle 402 identifies aspecific scenario occurring within the environment. The scenario mayinclude, but is not limited to, another vehicle merging into the drivelane of the vehicle 402, another vehicle navigating at a velocity thatis below a threshold velocity (which is described below), an accidentthat occurs along the route of the vehicle 402, and/or the like.

In some instances, the progress component 426 can further includefunctionality to determine when the progress of the vehicle 402 has beenimpeded for a threshold amount of time, which may be represented bythreshold data 444. Additionally, or alternatively, in some instances,the progress component 426 can include functionality to determine whenthe vehicle 402 is yielding to object(s) for a threshold amount of time.As described herein, the threshold amount of time may include, but isnot limited to, three seconds, thirty seconds, one minute, threeminutes, and/or any other period of time.

The conditions component 428 can include functionality to determine whenone or more intervening conditions are causing the progress of thevehicle 402 to be impeded. As described herein, an intervening conditionmay include, but is not limited to, the vehicle 402 being stuck intraffic, the vehicle 402 being impeded because of a traffic light,and/or the like.

The teleoperator component 430 can include functionality to determinewhen to contact the teleoperator. In some instances, the teleoperatorcomponent 430 determines to contact the teleoperator when the progressof the vehicle 402 has been impeded for the threshold amount of time. Insome instances, the teleoperator component 430 determines to contact theteleoperator when the progress of the vehicle 402 has been impeded forthe threshold amount of time and an intervening condition that causesthe progress of the vehicle 402 to be impeded has not occurred. Still,in some instances, the teleoperator component 430 determines to contactthe teleoperator when the progress of the vehicle has been impeded by aspecific type of object or specific scenario. In either instance, afterdetermining to contact the teleoperator, the teleoperator component 430can include functionality to cause the vehicle 402 to send at least aportion of the sensor data 436 to the teleoperator system 114.Additionally, in some instances, the teleoperator component 430 caninclude functionality to cause the vehicle 402 to send the indicatordata 442 to the teleoperator system 114.

In at least one example, the vehicle computing device 404 can includeone or more system controllers 432, which can be configured to controlsteering, propulsion, braking, safety, emitters, communication, andother systems of the vehicle 402. These system controller(s) 432 cancommunicate with and/or control corresponding systems of the drivemodule(s) 212 and/or other components of the vehicle 402.

The memory 418 can further include one or more maps 434 that can be usedby the vehicle 402 to navigate within the environment. For the purposeof this discussion, a map can be any number of data structures modeledin two dimensions, three dimensions, or N-dimensions that are capable ofproviding information about an environment, such as, but not limited to,topologies (such as intersections), streets, mountain ranges, roads,terrain, and the environment in general. In some instances, a map caninclude, but is not limited to: texture information (e.g., colorinformation (e.g., RGB color information, Lab color information, HSV/HSLcolor information), and the like), intensity information (e.g., lidarinformation, radar information, and the like); spatial information(e.g., image data projected onto a mesh, individual “surfels” (e.g.,polygons associated with individual color and/or intensity)),reflectivity information (e.g., specularity information,retroreflectivity information, BRDF information, BSSRDF information, andthe like). In one example, a map can include a three-dimensional mesh ofthe environment. In some instances, the map can be stored in a tiledformat, such that individual tiles of the map represent a discreteportion of an environment and can be loaded into working memory asneeded. In at least one example, the one or more maps 434 can include atleast one map (e.g., images and/or a mesh). In some example, the vehicle402 can be controlled based at least in part on the map(s) 434. That is,the map(s) 434 can be used in connection with the localization component420, the perception component 422, and/or the planning component 424 todetermine a location of the vehicle 402, identify entities in anenvironment, and/or generate routes and/or trajectories to navigatewithin an environment.

In some instances, aspects of some or all of the components discussedherein can include any models, algorithms, and/or machine learningalgorithms. For example, in some instances, the components in the memory418 can be implemented as a neural network. As described herein, anexemplary neural network is a biologically inspired algorithm whichpasses input data through a series of connected layers to produce anoutput. Each layer in a neural network can also comprise another neuralnetwork, or can comprise any number of layers (whether convolutional ornot). As can be understood in the context of this disclosure, a neuralnetwork can utilize machine learning, which can refer to a broad classof such algorithms in which an output is generated based at least inpart on learned parameters.

Although discussed in the context of neural networks, any type ofmachine learning can be used consistent with this disclosure. Forexample, machine learning algorithms can include, but are not limitedto, regression algorithms (e.g., ordinary least squares regression(OLSR), linear regression, logistic regression, stepwise regression,multivariate adaptive regression splines (MARS), locally estimatedscatterplot smoothing (LOESS)), instance-based algorithms (e.g., ridgeregression, least absolute shrinkage and selection operator (LASSO),elastic net, least-angle regression (LARS)), decisions tree algorithms(e.g., classification and regression tree (CART), iterative dichotomiser2 (ID2), Chi-squared automatic interaction detection (CHAID), decisionstump, conditional decision trees), Bayesian algorithms (e.g., naïveBayes, Gaussian naïve Bayes, multinomial naïve Bayes, averageone-dependence estimators (AODE), Bayesian belief network (BNN),Bayesian networks), clustering algorithms (e.g., k-means, k-medians,expectation maximization (EM), hierarchical clustering), associationrule learning algorithms (e.g., perceptron, back-propagation, hopfieldnetwork, Radial Basis Function Network (RBFN)), deep learning algorithms(e.g., Deep Boltzmann Machine (DBM), Deep Belief Networks (DBN),Convolutional Neural Network (CNN), Stacked Auto-Encoders),Dimensionality Reduction Algorithms (e.g., Principal Component Analysis(PCA), Principal Component Regression (PCR), Partial Least SquaresRegression (PLSR), Sammon Mapping, Multidimensional Scaling (MDS),Projection Pursuit, Linear Discriminant Analysis (LDA), MixtureDiscriminant Analysis (MDA), Quadratic Discriminant Analysis (QDA),Flexible Discriminant Analysis (FDA)), Ensemble Algorithms (e.g.,Boosting, Bootstrapped Aggregation (Bagging), AdaBoost, StackedGeneralization (blending), Gradient Boosting Machines (GBM), GradientBoosted Regression Trees (GBRT), Random Forest), SVM (support vectormachine), supervised learning, unsupervised learning, semi-supervisedlearning, etc.

Additional examples of architectures include neural networks such asResNet70, ResNet101, VGG, DenseNet, PointNet, and the like.

As discussed above, in at least one example, the sensor system(s) 406can include lidar sensors, radar sensors, ultrasonic transducers, sonarsensors, location sensors (e.g., GPS, compass, etc.), inertial sensors(e.g., inertial measurement units (IMUs), accelerometers, magnetometers,gyroscopes, etc.), cameras (e.g., RGB, IR, intensity, depth, time offlight, etc.), microphones, wheel encoders, environment sensors (e.g.,temperature sensors, humidity sensors, light sensors, pressure sensors,etc.), etc. The sensor system(s) 406 can include multiple instances ofeach of these or other types of sensors. For instance, the lidar sensorscan include individual lidar sensors located at the corners, front,back, sides, and/or top of the vehicle 402. As another example, thecamera sensors can include multiple cameras disposed at variouslocations about the exterior and/or interior of the vehicle 402. Thesensor system(s) 406 can provide input to the vehicle computing device404. Additionally or alternatively, the sensor system(s) 406 can sendthe sensor data 436, via the one or more network(s) 450, to a controlsystem 448 at a particular frequency, after a lapse of a predeterminedperiod of time, upon occurrence of one or more conditions, in nearreal-time, etc.

The vehicle 402 can also include one or more emitters 408 for emittinglight and/or sound, as described above. The emitter(s) 408 in thisexample include interior audio and visual emitters to communicate withpassengers of the vehicle 402. By way of example and not limitation,interior emitters can include speakers, lights, signs, display screens,touch screens, haptic emitters (e.g., vibration and/or force feedback),mechanical actuators (e.g., seatbelt tensioners, seat positioners,headrest positioners, etc.), and the like. The emitter(s) 408 in thisexample also include exterior emitters. By way of example and notlimitation, the exterior emitters in this example include lights tosignal a direction of travel or other indicator of vehicle action (e.g.,indicator lights, signs, light arrays, etc.), and one or more audioemitters (e.g., speakers, speaker arrays, horns, etc.) to audiblycommunicate with pedestrians or other nearby vehicles, one or more ofwhich comprising acoustic beam steering technology.

The vehicle 402 can also include one or more communication connection(s)410 that enable communication between the vehicle 402 and one or moreother local or remote computing device(s). For instance, thecommunication connection(s) 410 can facilitate communication with otherlocal computing device(s) on the vehicle 402 and/or the drive module(s)414. Also, the communication connection(s) 410 can allow the vehicle 402to communicate with other nearby computing device(s) (e.g., other nearbyvehicles, traffic signals, etc.). The communications connection(s) 410also enable the vehicle 402 to communicate with the remoteteleoperations computing devices (e.g., the teleoperator system 114) orother remote services.

The communications connection(s) 410 can include physical and/or logicalinterfaces for connecting the vehicle computing device 404 to anothercomputing device or a network, such as network(s) 450. For example, thecommunications connection(s) 410 can enable Wi-Fi-based communicationsuch as via frequencies defined by the IEEE 802.11 standards, shortrange wireless frequencies such as Bluetooth®, cellular communication(e.g., 2G, 2G, 4G, 4G LTE, 5G, etc.) or any suitable wired or wirelesscommunications protocol that enables the respective computing device tointerface with the other computing device(s).

In at least one example, the vehicle 402 can include one or more drivemodules 414. In some instances, the vehicle 402 can have a single drivemodule 414. In at least one example, if the vehicle 402 has multipledrive modules 414, individual drive modules 414 can be positioned onopposite ends of the vehicle 402 (e.g., the front and the rear, etc.).In at least one example, the drive module(s) 414 can include one or moresensor systems to detect conditions of the drive module(s) 414 and/orthe surroundings of the vehicle 402. By way of example and notlimitation, the sensor system(s) 406 can include one or more wheelencoders (e.g., rotary encoders) to sense rotation of the wheels of thedrive modules, inertial sensors (e.g., inertial measurement units,accelerometers, gyroscopes, magnetometers, etc.) to measure orientationand acceleration of the drive module, cameras or other image sensors,ultrasonic sensors to acoustically detect entities in the surroundingsof the drive module, lidar sensors, radar sensors, etc. Some sensors,such as the wheel encoders can be unique to the drive module(s) 414. Insome cases, the sensor system(s) 406 on the drive module(s) 414 canoverlap or supplement corresponding systems of the vehicle 402 (e.g.,sensor system(s) 406).

The drive module(s) 414 can include many of the vehicle systems,including a high voltage battery, a motor to propel the vehicle 402, aninverter to convert direct current from the battery into alternatingcurrent for use by other vehicle systems, a steering system including asteering motor and steering rack (which can be electric), a brakingsystem including hydraulic or electric actuators, a suspension systemincluding hydraulic and/or pneumatic components, a stability controlsystem for distributing brake forces to mitigate loss of traction andmaintain control, an HVAC system, lighting (e.g., lighting such ashead/tail lights to illuminate an exterior surrounding of the vehicle),and one or more other systems (e.g., cooling system, safety systems,onboard charging system, other electrical components such as a DC/DCconverter, a high voltage j unction, a high voltage cable, chargingsystem, charge port, etc.). Additionally, the drive module(s) 414 caninclude a drive module controller which can receive and preprocess thesensor data 436 from the sensor system(s) 406 and to control operationof the various vehicle systems. In some instances, the drive modulecontroller can include one or more processors and memory communicativelycoupled with the one or more processors. The memory can store one ormore modules to perform various functionalities of the drive module(s)414. Furthermore, the drive module(s) 414 also include one or morecommunication connection(s) that enable communication by the respectivedrive module with one or more other local or remote computing device(s).

In at least one example, the direct connection 412 can provide aphysical interface to couple the one or more drive module(s) 414 withthe body of the vehicle 402. For example, the direct connection 412 canallow the transfer of energy, fluids, air, data, etc. between the drivemodule(s) 414 and the vehicle 402. In some instances, the directconnection 412 can further releasably secure the drive module(s) 414 tothe body of the vehicle 402.

As further illustrated in FIG. 4, the control system 448 can includeprocessor(s) 452, communication connection(s) 454, and memory 456. Theprocessor(s) 416 of the vehicle 402 and/or the processor(s) 452 of thecontrol system 448 (and/or other processor(s) described herein) can beany suitable processor capable of executing instructions to process dataand perform operations as described herein. By way of example and notlimitation, the processor(s) 416 and 452 can comprise one or moreCentral Processing Units (CPUs), Graphics Processing Units (GPUs), orany other device or portion of a device that processes electronic datato transform that electronic data into other electronic data that can bestored in registers and/or memory. In some instances, integratedcircuits (e.g., ASICs, etc.), gate arrays (e.g., FPGAs, etc.), and otherhardware devices can also be considered processors in so far as they areconfigured to implement encoded instructions.

The memory 418 and the memory 456 (and/or other memory described herein)are examples of non-transitory computer-readable media. The memory 418and the memory 456 can store an operating system and one or moresoftware applications, instructions, programs, and/or data to implementthe methods described herein and the functions attributed to the varioussystems. In various implementations, the memory can be implemented usingany suitable memory technology, such as static random access memory(SRAM), synchronous dynamic RAM (SDRAM), nonvolatile/Flash-type memory,or any other type of memory capable of storing information. Thearchitectures, systems, and individual elements described herein caninclude many other logical, programmatic, and physical components, ofwhich those shown in the accompanying figures are merely examples thatare related to the discussion herein.

It should be noted that while FIG. 4 is illustrated as a distributedsystem, in alternative examples, components of the vehicle 402 can beassociated with the control system 448 and/or components of the controlsystem 448 can be associated with the vehicle 402. That is, the vehicle402 can perform one or more of the functions associated with the controlsystem 448, and vice versa.

FIG. 5 shows an example architecture 500 including a fleet of vehicles502 and the teleoperator system 114. The example vehicle fleet 502includes one or more vehicles (e.g., the vehicle 402), at least somewhich are communicatively coupled to the teleoperator system 114 via thecommunication connection(s) of the vehicles. A teleoperations receiver504 and a teleoperations transmitter 506 associated with theteleoperator system 114 may be communicatively coupled to the respectivecommunication connection(s) of the vehicles. For example, the vehicle402 may send communication signals via the communication connection(s)410, which are received by the teleoperations receiver 504. In someexamples, the communication signals may include, for example, the sensordata 436, the indicator data 442, operation state data, and/or any dataand/or output from one or more components of the vehicle 402 m. In someexamples, the sensor data 436 may include raw sensor data and/orprocessed sensor data, such as a subset of operation state data such asa representation of the sensor data (e.g. a bounding box). In someexamples, the communication signals may include a subset of theoperation state data discussed above and/or any other derived data(planning controls, etc.). In some examples, the communication signalsfrom the vehicle 402 may include a request for assistance to theteleoperator system 114.

As shown in FIG. 5, a situational awareness engine (“SAE”) 508 mayobtain the communication signals from the vehicle 402 and relay at leasta subset of them to a teleoperations interface 510. The SAE 508 mayadditionally or alternatively obtain signals generated via theteleoperations interface 510 and relay them to a teleoperations network512 and/or transmit at least a subset of them to one or more vehicles ofthe vehicle fleet 502 via the teleoperations transmitter 506. The SAE508 may additionally or alternatively obtain signals from theteleoperations network 512 and relay at least a subset of them to theteleoperations interface 510 and/or the teleoperations transmitter 506.In some examples, the teleoperations interface 510 may directlycommunicate with the vehicle fleet 502.

In some examples, the SAE 508 can be implemented on a device that isseparate from a device that includes the teleoperations interface 510.For example, the SAE 508 can include a gateway device and an applicationprogramming interface (“API”) or similar interface. In some examples,the SAE 508 includes an application interface and/or a model, such as,for example, a FSM, an ANN, and/or a DAG. In some examples, the SAE 508is configured to determine a presentation configuration of data receivedfrom one or more elements discussed herein (e.g., the vehicle 402, thevehicle fleet 502, and/or other teleoperations interfaces) and/or inputoptions to present to the teleoperator to provide guidance to one ormore vehicles (e.g., an option to select a displayed button thatconfirms a trajectory determined by the vehicle).

In some examples, the teleoperations receiver 504 may be communicativelycoupled to the teleoperations interface 510 via the SAE 508, and in someexamples, a teleoperator 514 may be able to access the sensor data 436,the indicator data 442, the operation state data, and/or any other datain the communication signals received from the vehicle 402 via theteleoperations interface 510. In some examples, the teleoperator 514 maybe able to selectively access the sensor data 436, the indicator data442, the operation state data, and/or other data via an input device,and view the selected data via one or more displays. In some examples,such selective accessing can include transmitting a request for datafrom the vehicle 402 via the teleoperations transmitter 506. In someexamples, the SAE 508 may present a subset or representation of the datato the teleoperator 514 via the teleoperations interface 510. As anon-limiting example, the SAE 508 may create simplistic pictorialrepresentations, bounding boxes, arrows indicating a bearing andvelocity of objects, icons representing objects, colorization of thesensor data, or other representations of the data which may simplifyinterpretation by a teleoperator 514.

In the example shown, the teleoperator system 114 also includes theteleoperations network 512 configured to provide communication betweentwo or more of the teleoperations interfaces 510 and the respectiveteleoperators 514, and/or communication with teleoperations data 516.For example, the teleoperator system 114 may include a plurality ofteleoperations interfaces 510 and respective teleoperators 514, and theteleoperators 514 may communicate with one another via theteleoperations network 512 to facilitate and/or coordinate the guidanceprovided to the vehicle fleet 502. In some examples, there may be ateleoperator 514 assigned to each vehicle from the vehicle fleet 502,and in some examples, a teleoperator 514 may be assigned to more than asingle vehicle of the vehicle fleet 502. In some examples, more than oneteleoperator 514 may be assigned to a single vehicle. In some examples,teleoperators 514 may not be assigned to specific vehicles of thevehicle fleet 502, but may instead provide guidance to vehicles thathave encountered certain types of events and/or to vehicles based atleast in part on, for example, a level of urgency associated with thevehicle's encounter with the event. In some examples, data associatedwith an event and/or the guidance provided by a teleoperator 514 may bestored by the teleoperator system 114, for example, in storage for theteleoperations data 516, and/or accessed by other teleoperators 514.

In some examples, the teleoperations data 516 may be accessible by theteleoperators 514, for example, via the teleoperations interface 510,for use in providing guidance to the vehicles 402. For example, theteleoperations data 516 may include global and/or local map data relatedto the road network, events associated with the road network, and/ortravel conditions associated with the road network due to, for example,traffic volume, weather conditions, construction zones, and/or specialevents. In some examples, the teleoperations data 516 may include dataassociated with one more of the vehicles of the vehicle fleet 502, suchas, for example, maintenance and service information, and/or operationalhistory including, for example, event history associated with thevehicle 402, route histories, occupancy histories, and other types ofdata associated with the vehicle 402.

In some examples, a teleoperator 514 and/or a teleoperations interface510 can be associated with credentials 518. For example, to activate asession at the teleoperations interface 510, the teleoperator system 114may require the teleoperator 514 to authenticate using the credentials518. In some examples, the credentials 518 may be inherent to theparticular teleoperations interface 510. In some examples, requests forassistance with a particular permutation of operation state data (e.g.,certain events) and/or a particular teleoperation option may requireresolution by a teleoperations interface 510 having elevated credentials518 (e.g., credentials with greater permissions). For example, if ateleoperator 514 selects an action at a teleoperations interface 510that would affect the entire vehicle fleet 502 instead of just onevehicle, the action could be transmitted to a second teleoperationsinterface 510 that has elevated credentials associated therewith forconfirmation, modification, and/or rejection. A request and/or operationstate data associated with an elevated level of credentials can be usedto determine a teleoperations interface 510 to which to relay therequest. In some examples, the SAE 508 can relay a request and/oroperation state data associated with an elevated level of credentials tomultiple teleoperations interfaces 510 instead of a singleteleoperations interface 510.

FIG. 6 is a first example user interface 602 showing the vehicle 402yielding for object(s), in accordance with embodiments of thedisclosure. As shown, the user interface 602 includes four portions604(1)-(4) illustrating information about the environment for which thevehicle 402 is located. For instance, the first portion 604(1) of theuser interface 602 includes a first video rendered as if taken fromabove the vehicle 402, the second portion 604(2) of the user interface602 includes a second video rendered as if taken from a side of thevehicle 402, the third portion 604(3) of the user interface 602 includesa third video captured by a first camera of the vehicle 402, and thefourth portion 604(4) of the user interface 602 includes a fourth videocaptured by a second camera of the vehicle 402. While the example ofFIG. 6 illustrates the user interface 602 as including four portions604(1)-(4), in other examples, the user interface 602 may include anynumber of portions.

As shown, the portions 604(1)-(2) of the user interface 602 includerepresentations of the vehicle 402. In some instances, one or more ofthe portions 604(1)-(2) of the user interface 602 may include graphicalelement(s) indicating the vehicle 402. The graphical element(s) mayinclude, but are not limited to, a bounding box around the vehicle 402,a shading of the vehicle 402, text describing the vehicle 402, and/orany other type graphical element that may indicate the vehicle 402. Thisway, the teleoperator 514 can easily identify where the vehicle 402 islocated within the environment.

Additionally, the portions 604(1)-(4) of the user interface 602 includerepresentations of the object 606 for which the vehicle 402 is yielding,which includes a crane parked at least partially within the intersectionin the example of FIG. 6. In some instances, one or more of the portions604(1)-(4) of the user interface 602 may include graphical element(s)indicating the object 606. The graphical element(s) may include, but arenot limited to, a bounding box around the object 606, a shading of theobject 606, text describing the object 606, and/or any other typegraphical element that may indicate the object 606. This way, theteleoperator 514 can easily identify the object 606 for which thevehicle 402 is yielding.

In some instances, the teleoperator 514 can select the object 606 usingthe user interface 602. The teleoperator 514 can then use the userinterface 602 to provide instructions for how the vehicle 402 is toproceed. For example, the teleoperator 514 can use the user interface602 to instruct the vehicle 402 to no longer yield to the object 606.This is because, in the example of FIG. 6, the object 606 parked atleast partially within the intersection and as such, is blocking theroute of the vehicle 402. Therefore, the vehicle 402 may have to ceaseyielding to the object 606 in order to navigate around the object 606.

The vehicle 402 may receive the instructions from the teleoperatorsystem 114 and proceed according to the instructions. For example, thevehicle 402 may determine to cease yielding to the object 606 and/ordetermine to temporarily cease following one or more policies withregard to the object 606. The vehicle 402 may then determine how toproceed navigating based on the one or more policies with regard toother objects located within the environment. For example, if thevehicle 402 is not yielding for another object, the vehicle 402 may thenselect a route such that the vehicle 402 navigates around the object606. The vehicle 402 may then begin to navigate along the selectedroute. However, if the vehicle 402 is still yielding to another object,then the vehicle 402 may continue to yield for the other object beforeproceeding.

For example, in some instances, the vehicle 402 may be yielding to morethan one object. For example, the vehicle 402 may further be yieldingfor a second object 608 that is located within the intersection. In suchan example, the user interface 602 may include a graphical elementindicating the second object 608. In some instances, the graphicalelement indicating the object 606 may be different than the graphicalelement indicating the second object 608. For example, the graphicalelement indicating the object 606 may tell the teleoperator 514 that thevehicle 402 is currently yielding for the object 606, and the graphicalelement indicating the second object 608 may tell the teleoperator 514that the vehicle 402 will later be yielding for the second object 608(e.g., if the second object 608 is still located within theintersection).

FIG. 7 is a first example user interface 702 showing the vehicle 402yielding for object(s), in accordance with embodiments of thedisclosure. As shown, the user interface 702 includes four portions704(1)-(4) illustrating information about the environment for which thevehicle 402 is located. For instance, the first portion 704(1) of theuser interface 702 includes a first video from above the vehicle 402,the second portion 704(2) of the user interface 702 includes a secondvideo from a side of the vehicle 402, the third portion 704(3) of theuser interface 702 includes a third video captured by a first camera ofthe vehicle 402, and the fourth portion 704(4) of the user interface 702includes a fourth video captured by a second camera of the vehicle 402.While the example of FIG. 7 illustrates the user interface 702 asincluding four portions 704(1)-(4), in other examples, the userinterface 702 may include any number of portions.

As shown, the portions 704(1)-(2) of the user interface 702 includerepresentations of the vehicle 402. In some instances, one or more ofthe portions 704(1)-(2) of the user interface 702 may include graphicalelement(s) indicating the vehicle 402. The graphical element(s) mayinclude, but are not limited to, a bounding box around the vehicle 402,a shading of the vehicle 402, text describing the vehicle 402, and/orany other type graphical element that may indicate the vehicle 402. Thisway, the teleoperator 514 can easily identify where the vehicle 402 islocated within the environment.

Additionally, the portions 704(1)-(3) of the user interface 702 includerepresentations of the object 706 for which the vehicle 402 is yielding,which includes a moving truck trying to navigate past the vehicle 402 inthe example of FIG. 7. In some instances, one or more of the portions704(1)-(3) of the user interface 702 may include graphical element(s)indicating the object 706. The graphical element(s) may include, but arenot limited to, a bounding box around the object 706, a shading of theobject 706, text describing the object 706, and/or any other typegraphical element that may indicate the object 706. This way, theteleoperator 514 can easily identify the object 706 for which thevehicle 402 is yielding.

In the example of FIG. 7, the vehicle 402 may be yielding to the object706. However, the object 706, which is attempting to navigate in thelane next to the vehicle 402, may be unable to navigate since an object708 is blocking at least a portion of the lane for which the object 706is using. As such, the object 706 may be stopped in the intersection andwaiting for the vehicle 402 to move in order for the object 706 tonavigate around the object 708. Additionally, the vehicle 402 may bestopped and yielding for the object 706 such that the vehicle 402 willnot proceed until the object 706 navigates through the intersection.

As such, and in some instances, the teleoperator 514 can select theobject 706 using the user interface 702. The teleoperator 514 can thenuse the user interface 702 to provide instructions for how the vehicle402 is to proceed. For example, the teleoperator 514 can use the userinterface 702 to instruct the vehicle 402 to no longer yield to theobject 706. This way, the vehicle 402 can begin to navigate even whenthe object 706 is still located within the intersection. Once thevehicle 402 begins to navigate, the object 706 will then also begin tonavigate around the object 708.

FIGS. 8-9 illustrate example processes in accordance with embodiments ofthe disclosure. These processes are illustrated as logical flow graphs,each operation of which represents a sequence of operations that can beimplemented in hardware, software, or a combination thereof. In thecontext of software, the operations represent computer-executableinstructions stored on one or more computer-readable storage media that,when executed by one or more processors, perform the recited operations.Generally, computer-executable instructions include routines, programs,objects, components, data structures, and the like that performparticular functions or implement particular abstract data types. Theorder in which the operations are described is not intended to beconstrued as a limitation, and any number of the described operationscan be combined in any order and/or in parallel to implement theprocesses.

FIG. 8 depicts an example process 800 for determining whether to contacta teleoperator, in accordance with embodiments of the presentdisclosure. At operation 802, the process 800 may include causing anautonomous vehicle to navigate along a route. For instance, the vehicle402 may receive instructions from the control system 448, where theinstructions cause the vehicle 402 to navigate along the route from afirst location to a second location. Based at least in part on theinstructions, the vehicle 402 may begin navigating along the route fromthe first location to the second location.

At operations 804, the process 800 may include determining that progressof the autonomous vehicle is impeded at a location along the route. Forinstance, the vehicle 402 may determine that the progress of the vehicle402 is impeded at the location along the route. In some instances, thevehicle 402 may make the determination based on determining that thevehicle is yielding to object(s) for a threshold period of time. In someinstances, the vehicle 402 may make the determination based ondetermining that a specific scenario is occurring around the vehicle402. Still, in some instances, the vehicle 402 may make thedetermination based on determining that a specific type of object islocated near the vehicle 402.

At operation 806, the process 800 may include determining an amount oftime that the progress of the autonomous vehicle has been impeded. Forinstance, the vehicle 402 may determine the amount of time at which theprogress of the vehicle 402 has been impeded.

At operations 808, the process 800 may include determining if the amountof time satisfies a threshold. For instance, the vehicle 402 maydetermine if the amount of time satisfies (e.g., is equal to or greaterthan) the threshold. If, at operation 808, the amount of time does notsatisfy the threshold, then at operation 810, the process 800 mayinclude determining not to contact a remote operator. For instance, ifthe vehicle 402 determines that the amount of time does not satisfy(e.g., if below) the threshold, then the vehicle 402 may determine notto contact the teleoperator 514. Additionally, the vehicle 402 mayrepeat operations 802-808.

However, if at operation 808, the amount of time satisfies thethreshold, then at operation 812, the process 800 may includedetermining if an intervening condition has occurred. For instance, thevehicle 402 may determine if an intervening condition is causing theprogress to be impeded. In some instances, the vehicle 402 may determinethat the intervening condition has occurred when the vehicle 402 isstuck in traffic. In some instances, the vehicle 402 may determine thatthe intervening condition has occurred when the vehicle 402 is stoppedat a traffic light.

If, at operation 812, the intervening condition has occurred, then theprocess 800 may determine not to contact the remote operator. Forinstance, if the vehicle 402 determines that the intervening conditionhas occurred, then the vehicle 402 may determine not to contact theteleoperator 514.

However, if at operation 812, the interviewing condition has notoccurred, then at operation 814, the process 800 may include determiningto contact the remote operator. For instance, if the vehicle 402determines that the intervening condition has not occurred, then thevehicle 402 may determine to contact the teleoperator 514. Additionally,the vehicle 402 may send sensor data 436 and/or indicator data 442 tothe teleoperator system 114. The teleoperator 514 may then use thesensor data 436 and/or indicator data 442 to determine how the vehicle402 should proceed.

At operation 818, the process 800 may include receiving, from a system,an instruction associated with navigating. For instance, the vehicle 402may receive instruction data 446 from the teleoperator system 114, wherethe instruction data 446 represents an instruction associated withnavigating the vehicle 402. For a first example, the instruction data446 may include an instruction to temporarily cease following the policyassociated with yielding to objects that arrive at intersections beforethe vehicle 402. For a second example, the instruction data 446 mayinclude an instruction to change an interaction with the object from“follow” to “do not follow”. In either example, the vehicle 402 maycease yielding to the object.

At operation 820, the process 800 may include causing the autonomousvehicle to navigate. For instance, based at least in part on theinstruction data 446, the vehicle 402 may begin navigating. In someinstances, the vehicle 402 begins navigating since the vehicle 402 is nolonger yielding to object(s). In some instances, the vehicle 402 beginsnavigating since the vehicle is no longer following object(s). Still, insome instances, the vehicle 402 begins navigating along the originalroute of the vehicle.

FIG. 9 depicts an example process 900 for navigating an autonomousvehicle using instructions from a teleoperator, in accordance withembodiments of the present disclosure. At operation 902, the process 900may include generating sensor data using one or more sensors. Forinstance, the vehicle 402 may use the sensor system(s) 406 to generatethe sensor data 436. The vehicle 402 may generate the sensor data 436while navigating around an environment.

At operations 904, the process 900 may include determining that progressof the autonomous vehicle is impeded. For instance, the vehicle 402 maydetermine that the progress of the vehicle 402 is impeded. In someinstances, the vehicle 402 may make the determination based ondetermining that the vehicle is yielding to object(s) for a thresholdperiod of time. In some instances, the vehicle 402 may make thedetermination based on determining that a specific scenario is occurringaround the vehicle 402. Still, in some instances, the vehicle 402 maymake the determination based on determining that a specific type ofobject is located near the vehicle 402.

At operation 906, the process 900 may include sending at least a portionof the sensor data to a teleoperator system (and/or data derivedtherefrom). For instance, the vehicle 402 may send the at least theportion of the sensor data 436 to the teleoperator system 114. The atleast the portion of the sensor data 436 may represent at least theobject causing the progress to be impeded. In some instances, thevehicle 402 sends the at least the portion of the sensor data 436 basedat least in part on determining that progress of the vehicle 402 hasbeen impeded for a threshold amount of time.

At operation 908, the process 900 may include sending, to theteleoperator system, an indication that the progress of the autonomousvehicle is impeded to an object. For instance, the vehicle 402 may sendindicator data 442 to the teleoperator system 114, where the indicatordata 442 indicates that the progress of the vehicle 402 is impeded tothe object. In some instances, the indicator data 442 may furtherindicate the amount of time that the vehicle 402 has been impeded to theobject.

At operation 910, the process 800 may include receiving, from theteleoperator system, an instruction associated with navigating. Forinstance, the vehicle 402 may receive instruction data 446 from theteleoperator system 114, where the instruction data 446 represents aninstruction associated with navigating the vehicle 402. For a firstexample, the instruction data 446 may include an instruction totemporarily cease following the policy associated with yielding toobjects that arrive at intersections before the vehicle 402. For asecond example, the instruction data 446 may include an instruction tochange an interaction with the object from “follow” to “do not follow”.In either example, the vehicle 402 may cease yielding to the object.

At operation 912, the process 900 may include causing the autonomousvehicle to navigate to a second location. For instance, based at leastin part on the instruction data 446, the vehicle 402 may beginnavigating to the second location. In some instances, the vehicle 402begins navigating since the vehicle 402 is no longer yielding toobject(s). In some instances, the vehicle 402 begins navigating sincethe vehicle is no longer following object(s). Still, in some instances,the vehicle 402 begins navigating along the original route of thevehicle.

At operation 914, the process 900 may include determining if anintervening condition occurs while navigating to the second location.For instance, the vehicle 402 may use the sensor data 436 to determineif the intervening condition occurs after beginning to navigate to thesecond location. In some instances, the intervening condition mayinclude the object beginning to move. In some instances, the interveningcondition may include the object moving outside of a designated areaassociated with the object.

If, at operation 914, the intervening condition occurs, then atoperation 916, the process 900 may include causing the autonomousvehicle to cease navigating to the second location. For instance, if thevehicle 402 determines that the intervening condition occurs, then thevehicle 402 may cease navigating to the second location. In someinstances, the vehicle 402 may stop. In some instances, the vehicle 402may change an interaction with the object from “do not follow” to“follow”.

However, if at operation 914, the intervening condition does not occur,then at operation 918, the process 900 may include causing theautonomous vehicle to continue navigating to the second location. Forinstance, if the vehicle 402 determines that the intervening conditiondid not occur, then the vehicle 402 will continue to navigate to thesecond location.

EXAMPLE CLAUSES

A: An autonomous vehicle comprising: one or more network components; oneor more sensors; one or more processors; and memory storing instructionsthat, when executed by the one or more processors, configure theautonomous vehicle to perform operations comprising: causing theautonomous vehicle to navigate along a route; receiving sensor datagenerated by the one or more sensors; determining, based at least inpart on the sensor data, that a progress of the autonomous vehicle isimpeded by an object; determining, based at least in part on theprogress being impeded by the object, whether to contact a remoteoperator; receiving, from one or more computing devices associated withthe remote operator, an instruction associated with navigating by theobject comprising information configured to alter a policy associatedwith the object; and causing, based at least in part on the instruction,the autonomous vehicle to navigate by the object.

B: The autonomous vehicle as recited in paragraph A, the operationsfurther comprising sending, using the one or more network components: atleast a portion of the sensor data to the one or more computing devices;and an indication that the object is impeding the progress of theautonomous vehicle.

C: The autonomous vehicle as recited in either of paragraphs A or B,wherein the determining that the progress of the autonomous vehicle isimpeded by the object comprises determining, based at least in part onthe sensor data, that the autonomous vehicle is yielding to the objectfor a threshold period of time.

D: The autonomous vehicle as recited in any of paragraphs A-C, whereinthe determining that the progress of the autonomous vehicle is impededby the object comprises: determining, based at least in part on thesensor data, that the object includes a type of object; and determiningthat the object is located along the route.

E: The autonomous vehicle as recited in any of paragraphs A-D, whereinthe determining that the progress of the autonomous vehicle is impededcomprises determining, based at least in part on the sensor data, that ascenario is occurring, the scenario including at least one of: avelocity of the object being below a threshold velocity; or the objectswitching from a first drive lane to a second drive lane, wherein theautonomous vehicle is navigating in the second drive lane.

F: The autonomous vehicle as recited in any of paragraphs A-E, theoperations further comprising: determining whether an interveningcondition is occurring, the intervening condition including at least oneof: the autonomous vehicle being stuck in traffic; or the autonomousvehicle being stopped at a traffic light, and wherein the determiningwhether to contact the remote operator is further based at least in parton the determining whether the intervening condition is occurring.

G: A method comprising: receiving sensor data generated by one or moresensors of an autonomous vehicle; determining, based at least in part onthe sensor data, that progress of the autonomous vehicle is impeded byan object; based at least in part on the progress of the autonomousvehicle being impeded by the object, determining whether to contact aremote operator; and receiving, from the remote operator, one or moreinstructions configured to cause the autonomous vehicle to alter apolicy with respect to the object.

H: The method as recited in paragraph G, further comprising sending atleast one of: at least a portion of the sensor data to one or morecomputing devices associated with the remote operator; or an indicationof the object to the one or more computing devices.

I: The method as recited in either of paragraphs G or H, wherein thedetermining that the progress of the autonomous vehicle is impeded bythe object comprises determining, based at least in part on the sensordata, that the autonomous vehicle is yielding to the object for athreshold period of time.

J: The method as recited in any of paragraphs G-I, wherein thedetermining that the progress of the autonomous vehicle is impeded bythe object comprises: determining, based at least in part on the sensordata, that the object includes a type of object; and determining thatthe object is located along the route.

K: The method as recited in any of paragraphs G-J, wherein thedetermining that the progress of the autonomous vehicle is impededcomprises determining, based at least in part on the sensor data, that ascenario is occurring, the scenario including at least one of: avelocity of the object being below a threshold velocity; or the objectswitching from a first drive lane to a second drive lane, wherein theautonomous vehicle is navigating in the second drive lane.

L: The method as recited in any of paragraphs G-K, further comprising:determining whether an intervening condition is occurring, theintervening condition including at least one of: the autonomous vehiclebeing stuck in traffic; or the autonomous vehicle being stopped at atraffic light, and wherein the determining whether to contact the remoteoperator is further based at least in part on the determining whetherthe intervening condition is occurring.

M: The method as recited in any of paragraphs G-L, further comprisingsending, to one or more computing devices associated with the remoteoperator, policy data indicating a policy that is causing the progressof the autonomous vehicle to be impeded by the object.

N: The method as recited in any of paragraphs G-M, further comprising:receiving, from one or more computing devices associated with the remoteoperator, an instruction to temporarily cease following the policy forthe object; and determining to temporarily cease following the policyfor the object.

O: A non-transitory computer-readable media storing instructions that,when executed by one or more processors, cause one or more computingdevices to perform operations comprising: receiving sensor datagenerated by one or more sensors of an autonomous vehicle; determining,based at least in part on the sensor data, that an object has causedprogress of the autonomous vehicle to be impeded; based at least in parton the object causing the progress of the autonomous vehicle to beimpeded, determining whether to contact a remote operator; andreceiving, from the remote operator, instructions configured to causethe autonomous vehicle to alter one or more policies associated with theobject.

P: The non-transitory computer-readable media as recited in paragraph O,the operations further comprising sending at least one of: at least aportion of the sensor data to one or more computing devices associatedwith the remote operator; or an indication of the object to the one ormore computing devices.

Q: The non-transitory computer-readable media as recited in either ofparagraphs O or P, wherein the determining that the object has causedthe progress of the autonomous vehicle to be impeded comprisesdetermining, based at least in part on the sensor data, that theautonomous vehicle is yielding to the object for a threshold period oftime.

R: The non-transitory computer-readable media as recited in any ofparagraphs O-Q, wherein the determining that the object has caused theprogress of the autonomous vehicle to be impeded comprises: determining,based at least in part on the sensor data, that the object includes atype of object; and determining that the object is located along theroute.

S: The non-transitory computer-readable media as recited in any ofparagraphs O-R, wherein the determining that the object has caused theprogress of the autonomous vehicle to be impeded comprises determining,based at least in part on the sensor data, that a scenario is occurring,the scenario including at least one of: a velocity of the object beingbelow a threshold velocity; or the object switching from a first drivelane to a second drive lane, wherein the autonomous vehicle isnavigating in the second drive lane.

T: The non-transitory computer-readable media as recited in any ofparagraphs O-S, the operations further comprising: determining whetheran intervening condition is occurring, the intervening conditionincluding at least one of: the autonomous vehicle being stuck intraffic; or the autonomous vehicle being stopped at a traffic light, andwherein the determining whether to contact the remote operator isfurther based at least in part on the determining whether theintervening condition is occurring.

U: An autonomous vehicle comprising: a sensor; one or more processors;and memory storing instructions that, when executed by the one or moreprocessors, configure the autonomous vehicle to perform operationscomprising: receiving sensor data from the sensor at a first location;determining, based at least in part on the sensor data, that progress ofthe autonomous vehicle is impeded by an object; transmitting at least aportion of the sensor data to one or more computing devices associatedwith a remote operator; transmitting an indication that the progress ofautonomous vehicle is impeded by the object; receiving an instructionassociated with navigating the autonomous vehicle; and based at least inpart on the instruction, causing the autonomous vehicle to beginnavigating from the first location to a second location along the route.

V: The autonomous vehicle as recited in paragraph U, the operationsfurther comprising: determining, while navigating to the secondlocation, that the object begins moving; and based at least in part onthe determining that the object begins moving, causing the autonomousvehicle to yield to the object.

W: The autonomous vehicle as recited in either of paragraphs U or V, theoperations further comprising: receiving, from the one or more computingdevices, an additional indication of an area that surrounds the object;determining, while navigating to the second location, that the object islocated outside of the area; and based at least in part on the objectbeing located outside of the area, causing the autonomous vehicle toyield to the object.

X: The autonomous vehicle as recited in any of paragraphs U-W, theoperations further comprising: storing data representing one or morepolicies associated with navigating the autonomous vehicle, the one ormore policies including at least one of: a first policy to yield toobjects that arrive at an intersection before the autonomous vehicle; ora second policy to stay in a driving lane when navigating; and based atleast in part on the instruction, determining to temporarily foregofollowing at least one of the one or more policies.

Y: The autonomous vehicle as recited in any of paragraphs U-X, theoperations further comprising: setting a first action associated withthe object, the first action being to follow the object; and based atleast in part on the instruction, setting a second action associatedwith the object, the second action being to cease following the object.

Z: The autonomous vehicle as recited in any of paragraphs U-Y, theoperations further comprising: determining, based at least in part onthe instruction, to temporarily cease yielding to the object; anddetermining, based at least in part on one or more policies, whether toyield to one or more other objects, and wherein the causing theautonomous vehicle to begin navigating from the first location to thesecond location along the route is based at least in part on determiningnot to yield to the one or more other objects.

AA: A method comprising: receiving sensor data from a sensor on avehicle; determining, based at least in part on the sensor data, thatprogress of the vehicle along a route is impeded by an object in theenvironment; determining whether to contact a remote operator;transmitting, based at least in part on determining whether to contactthe remote operator, one or more of at least a portion of the sensordata, a representation of the sensor data associated with the object, oran indication that the object is impeding progress to one or morecomputing devices associated with the remote operator; receiving, fromthe one or more computing devices, an instruction altering a policyassociated with the object; and based at least in part on theinstruction, causing the autonomous vehicle to navigate to a location.

AB: The method as recited in paragraph AA, wherein: the indicationcomprises data identifying that the vehicle is yielding to the object;and the altering the policy comprises instructing the vehicle to stopyielding to the object.

AC: The method as recited in either of paragraphs AA or AB, furthercomprising: determining, while navigating to the location, that theobject begins moving; and based at least in part on the determining thatthe object begins moving, causing the vehicle to yield to the object.

AD: The method as recited in any of paragraphs AA-AC, furthercomprising: receiving, from the one or more computing devices, an areathat surrounds the object; determining, while navigating to thelocation, that the object is located outside of the area; and based atleast in part on the object being located outside of the area, causingthe vehicle to yield to the object.

AE: The method as recited in any of paragraph AA-AD, wherein the policycomprises at least one of: a first policy to yield to objects thatarrive at an intersection before the vehicle, or a second policy to stayin a driving lane when navigating, and wherein altering the policycomprises determining to temporarily forego following at least one ofthe first policy or second policy.

AF: The method as recited in any of paragraphs AA-AE, furthercomprising, after navigating to the location, determining to againfollow the first or second policies.

AG: The method as recited in any of paragraphs AA-AF, furthercomprising: setting a first action associated with the object, the firstaction being to follow the object, wherein altering the policy comprisessetting a second action associated with the object, the second actionbeing to cease following the object.

AH: The method as recited in any of paragraphs AA-AG, further comprisingdetermining whether to impede progress of the autonomous vehicle toanother object based at least in part on the progress.

AI: One or more non-transitory computer-readable media storinginstructions that, when executed by one or more processors, cause one ormore computing devices to perform operations comprising: receivingsensor data from a sensor on a vehicle; determining, based at least inpart on the sensor data, that an object has caused the vehicle to beimpeded from navigating to a location; transmitting at least a portionof the sensor data to one or more computing devices associated with aremote operator; receiving, from the one or more computing devices, aninstruction to alter a policy with respect to the object; and based atleast in part on the instruction, causing the vehicle to navigate to thelocation.

AJ: The one or more non-transitory computer-readable media as recited inparagraph AI, the operations further comprising transmitting, to the oneor more computing devices, an indication that the object has caused theautonomous vehicle to be impeded from navigating to the location.

AK: The one or more non-transitory computer-readable media as recited ineither of paragraphs AI or AJ, the operations further comprising:determining, while navigating to the location, that the object beginsmoving; and based at least in part on the determining that the objectbegins moving, causing the vehicle to ignore the instruction.

AL: The one or more non-transitory computer-readable media as recited inany of paragraphs AI-AK, the operations further comprising: receiving,from the one or more computing devices, an area associated with theobject; determining, while navigating to the location, that the objectis located outside of the area; and based at least in part on the objectbeing located outside of the area, causing the autonomous vehicle toignore the instruction.

AM: The one or more non-transitory computer-readable media as recited inany of paragraphs AI-AL, wherein the policy comprises one or more of: afirst policy to yield to objects that arrive at an intersection beforethe autonomous vehicle, or a second policy to stay in a driving lanewhen navigating, and the operations further comprise, based at least inpart on the instruction, determining to temporarily forego following atleast one of the one or more policies.

AN: The one or more non-transitory computer-readable media as recited inany of paragraphs AI-AM, the operations further comprising: setting afirst action associated with the object, the first action being tofollow the object; and based at least in part on the instruction,setting a second action associated with the object, the second actionbeing to cease following the object.

Conclusion

While one or more examples of the techniques described herein have beendescribed, various alterations, additions, permutations and equivalentsthereof are included within the scope of the techniques describedherein.

In the description of examples, reference is made to the accompanyingdrawings that form a part hereof, which show by way of illustrationspecific examples of the claimed subject matter. It is to be understoodthat other examples can be used and that changes or alterations, such asstructural changes, can be made. Such examples, changes or alterationsare not necessarily departures from the scope with respect to theintended claimed subject matter. While the steps herein can be presentedin a certain order, in some cases the ordering can be changed so thatcertain inputs are provided at different times or in a different orderwithout changing the function of the systems and methods described. Thedisclosed procedures could also be executed in different orders.Additionally, various computations that are herein need not be performedin the order disclosed, and other examples using alternative orderingsof the computations could be readily implemented. In addition to beingreordered, the computations could also be decomposed intosub-computations with the same results.

What is claimed is:
 1. An autonomous vehicle comprising: one or morenetwork components; one or more sensors; one or more processors; andmemory storing instructions that, when executed by the one or moreprocessors, configure the autonomous vehicle to perform operationscomprising: causing the autonomous vehicle to navigate along a route;receiving sensor data generated by the one or more sensors; determining,based at least in part on the sensor data, that a progress of theautonomous vehicle is impeded by an object; determining, based at leastin part on the progress being impeded by the object, whether to contacta remote operator; receiving, from one or more computing devicesassociated with the remote operator, an instruction associated withnavigating by the object comprising information configured to alter apolicy associated with the object; and causing, based at least in parton the instruction, the autonomous vehicle to navigate by the object. 2.The autonomous vehicle as recited in claim 1, the operations furthercomprising sending, using the one or more network components: at least aportion of the sensor data to the one or more computing devices; and anindication that the object is impeding the progress of the autonomousvehicle.
 3. The autonomous vehicle as recited in claim 1, wherein thedetermining that the progress of the autonomous vehicle is impeded bythe object comprises determining, based at least in part on the sensordata, that the autonomous vehicle is yielding to the object for athreshold period of time.
 4. The autonomous vehicle as recited in claim1, wherein the determining that the progress of the autonomous vehicleis impeded by the object comprises: determining, based at least in parton the sensor data, that the object includes a type of object; anddetermining that the object is located along the route.
 5. Theautonomous vehicle as recited in claim 1, wherein the determining thatthe progress of the autonomous vehicle is impeded comprises determining,based at least in part on the sensor data, that a scenario is occurring,the scenario including at least one of: a velocity of the object beingbelow a threshold velocity; or the object switching from a first drivelane to a second drive lane, wherein the autonomous vehicle isnavigating in the second drive lane.
 6. The autonomous vehicle asrecited in claim 1, the operations further comprising: determiningwhether an intervening condition is occurring, the intervening conditionincluding at least one of: the autonomous vehicle being stuck intraffic; or the autonomous vehicle being stopped at a traffic light, andwherein the determining whether to contact the remote operator isfurther based at least in part on the determining whether theintervening condition is occurring.
 7. A method comprising: receivingsensor data generated by one or more sensors of an autonomous vehicle;determining, based at least in part on the sensor data, that progress ofthe autonomous vehicle is impeded by an object; based at least in parton the progress of the autonomous vehicle being impeded by the object,determining whether to contact a remote operator; and receiving, fromthe remote operator, one or more instructions configured to cause theautonomous vehicle to alter a policy with respect to the object.
 8. Themethod as recited in claim 7, further comprising sending at least oneof: at least a portion of the sensor data to one or more computingdevices associated with the remote operator; or an indication of theobject to the one or more computing devices.
 9. The method as recited inclaim 7, wherein the determining that the progress of the autonomousvehicle is impeded by the object comprises determining, based at leastin part on the sensor data, that the autonomous vehicle is yielding tothe object for a threshold period of time.
 10. The method as recited inclaim 7, wherein the determining that the progress of the autonomousvehicle is impeded by the object comprises: determining, based at leastin part on the sensor data, that the object includes a type of object;and determining that the object is located along the route.
 11. Themethod as recited in claim 7, wherein the determining that the progressof the autonomous vehicle is impeded comprises determining, based atleast in part on the sensor data, that a scenario is occurring, thescenario including at least one of: a velocity of the object being belowa threshold velocity; or the object switching from a first drive lane toa second drive lane, wherein the autonomous vehicle is navigating in thesecond drive lane.
 12. The method as recited in claim 7, furthercomprising: determining whether an intervening condition is occurring,the intervening condition including at least one of: the autonomousvehicle being stuck in traffic; or the autonomous vehicle being stoppedat a traffic light, and wherein the determining whether to contact theremote operator is further based at least in part on the determiningwhether the intervening condition is occurring.
 13. The method asrecited in claim 7, further comprising sending, to one or more computingdevices associated with the remote operator, policy data indicating apolicy that is causing the progress of the autonomous vehicle to beimpeded by the object.
 14. The method as recited in claim 13, furthercomprising: receiving, from one or more computing devices associatedwith the remote operator, an instruction to temporarily cease followingthe policy for the object; and determining to temporarily ceasefollowing the policy for the object.
 15. A non-transitorycomputer-readable media storing instructions that, when executed by oneor more processors, cause one or more computing devices to performoperations comprising: receiving sensor data generated by one or moresensors of an autonomous vehicle; determining, based at least in part onthe sensor data, that an object has caused progress of the autonomousvehicle to be impeded; based at least in part on the object causing theprogress of the autonomous vehicle to be impeded, determining whether tocontact a remote operator; and receiving, from the remote operator,instructions configured to cause the autonomous vehicle to alter one ormore policies associated with the object.
 16. The non-transitorycomputer-readable media as recited in claim 15, the operations furthercomprising sending at least one of: at least a portion of the sensordata to one or more computing devices associated with the remoteoperator; or an indication of the object to the one or more computingdevices.
 17. The non-transitory computer-readable media as recited inclaim 15, wherein the determining that the object has caused theprogress of the autonomous vehicle to be impeded comprises determining,based at least in part on the sensor data, that the autonomous vehicleis yielding to the object for a threshold period of time.
 18. Thenon-transitory computer-readable media as recited in claim 15, whereinthe determining that the object has caused the progress of theautonomous vehicle to be impeded comprises: determining, based at leastin part on the sensor data, that the object includes a type of object;and determining that the object is located along the route.
 19. Thenon-transitory computer-readable media as recited in claim 15, whereinthe determining that the object has caused the progress of theautonomous vehicle to be impeded comprises determining, based at leastin part on the sensor data, that a scenario is occurring, the scenarioincluding at least one of: a velocity of the object being below athreshold velocity; or the object switching from a first drive lane to asecond drive lane, wherein the autonomous vehicle is navigating in thesecond drive lane.
 20. The non-transitory computer-readable media asrecited in claim 15, the operations further comprising: determiningwhether an intervening condition is occurring, the intervening conditionincluding at least one of: the autonomous vehicle being stuck intraffic; or the autonomous vehicle being stopped at a traffic light, andwherein the determining whether to contact the remote operator isfurther based at least in part on the determining whether theintervening condition is occurring.